Skip to content

Include exception message in JUnit failure/error bodies - #10286

Merged
Evangelink merged 3 commits into
mainfrom
dev/amauryleve/bookish-bassoon
Jul 28, 2026
Merged

Include exception message in JUnit failure/error bodies#10286
Evangelink merged 3 commits into
mainfrom
dev/amauryleve/bookish-bassoon

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Fixes #10269

Include the exception message in <failure>/<error> bodies

Today the JUnit report writes only Exception.StackTrace into the element body, keeping the message in the message attribute. Consumers that render the body rather than the attribute — GitLab CI and CircleCI most notably — therefore show a bare stack trace with no indication of why the test failed. This is especially damaging for fluent assertion libraries, where the stack trace is mostly framework frames while the assertion message carries the actual diagnosis.

This is a fidelity gap, not a stylistic choice. Canonical JUnit bodies come from Java's Throwable.printStackTrace(), whose first line is com.example.AssertionError: expected:<1> but was:<2> — the message is inherently part of the body. .NET's Exception.StackTrace omits that header (only ToString() includes it), which is exactly why our output looked message-less next to a genuine Surefire report. GitLab isn't being quirky; it's reading the body assuming Java semantics.

The body now mirrors printStackTrace():

<failure message="Expected the operation not to throw, but it threw ..."
         type="Xunit.Sdk.XunitException">
Xunit.Sdk.XunitException: Expected the operation not to throw, but it threw ...
   at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
   ...
</failure>

Each part degrades gracefully: a missing exception type drops the header prefix, a missing message drops the : separator, and a missing stack trace yields a header-only body.

Why change the default instead of adding an option

The issue proposed a --report-junit-failure-body-format option. We went with changing the behavior outright:

  • Microsoft.Testing.Extensions.JUnitReport has not shipped a stable version (only 1.0.0-alpha.* on NuGet), so there is no back-compat cost to fixing the default now. This is the one moment where it's free.
  • A CLI option is a permanent contract, and it would only let users opt into less diagnostic information. Adding one later is non-breaking; removing one is not.
  • Both schemas allow it: neither the Ant/windyroad JUnit.xsd nor Surefire's surefire-test-report.xsd constrains the body — it's free-form text (the Ant schema's "e.g., a stack trace" is illustrative, not normative).

The message and type attributes are unchanged, so consumers reading them directly are unaffected. The resulting duplication between attribute and body is exactly what every Maven/Surefire report already exhibits, so consumers rendering both have long handled it.

Unlike spekt's FailureBodyFormat=Verbose, we deliberately do not fold standard output into the failure body — it's already emitted as <system-out> per test case, and duplicating it would bloat reports for noisier suites.

Always emit the type attribute

The Ant/windyroad JUnit.xsd marks type as use="required" on <failure>/<error> (Surefire relaxes it to optional). MTP only supplies an exception type when the state property carried an actual Exception — frameworks that report a failure through Explanation alone leave it null, so the attribute was previously omitted entirely in that case.

type is now always written, falling back to the element name (failure / error) when no exception type is available. That keeps the document valid under the stricter schema without inventing a bogus exception type name.

Testing

  • New unit tests (JUnitReportFailureBodyTests) covering every combination of present/absent exception type, message and stack trace, plus the type fallback for both <failure> and <error>, asserted against real generated XML.
  • Updated acceptance snapshot in Microsoft.Testing.Platform.Acceptance.IntegrationTests/JUnitReportTests.cs — the <failure>/<error> elements are no longer self-closing.
  • Verified locally: 795 unit tests pass, 21 MTP JUnit acceptance tests pass across all TFMs, and the MSTest JUnit retry acceptance test (which produces real stack traces) passes.

Docs

docs/RFCs/016-JUnit-Report.md updated with a "Failure and error body format" section, a "The type attribute" section, and the revised outcome-mapping table.

.NET's Exception.StackTrace omits the leading 	ype: message header that
Java's Throwable.printStackTrace() (and Exception.ToString()) include, so
writing the stack trace alone into the <failure>/<error> body dropped the
single most useful piece of diagnostic information. Consumers that render
the body rather than the message attribute -- GitLab CI and CircleCI most
notably -- showed only a stack trace with no indication of why a test failed.

The body now mirrors printStackTrace() shape, degrading gracefully when the
exception type, message or stack trace is absent. The message and 	ype
attributes are unchanged, so consumers reading them directly are unaffected.

Also always emit the 	ype attribute. The Ant/windyroad JUnit.xsd marks it
use="required" (Surefire relaxes it to optional), but MTP only supplies an
exception type when the state property carried an actual Exception --
frameworks reporting via Explanation alone leave it null. Fall back to the
element name so the document stays valid under the stricter schema.

Fixes #10269

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a565c557-8c36-4279-ab7b-41311abcd836
Copilot AI review requested due to automatic review settings July 28, 2026 08:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates JUnit reports to provide complete failure diagnostics in body-rendering CI systems.

Changes:

  • Adds exception type/message headers to failure and error bodies.
  • Always emits the required type attribute.
  • Adds unit, acceptance, API tracking, and RFC updates.
Show a summary per file
File Description
JUnitXmlWriter.cs Builds diagnostic failure/error bodies and type fallbacks.
InternalAPI.Unshipped.txt Tracks the new internal helper.
JUnitReportFailureBodyTests.cs Tests body composition and XML output.
JUnitReportTests.cs Updates acceptance snapshots.
016-JUnit-Report.md Documents the revised format.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Medium

Comment thread docs/RFCs/016-JUnit-Report.md Outdated
@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

🔴 Build Failure Analysis

Root cause: All 6 build legs fail with the same error in src/Analyzers/MSTest.Analyzers/MSTest.Analyzers.csprojnot in any file changed by this PR:

'xlf/Resources.fr.xlf' is out-of-date with 'Resources.resx'.
Run `msbuild /t:UpdateXlf` to update .xlf files

Why this PR is affected: The most recent commit to src/Analyzers/MSTest.Analyzers/Resources.resx on main was #10248 (2026-07-27), which added new analyzer resource strings but did not regenerate the .xlf files. This PR was branched after or rebased onto that commit, so it inherits the broken base.

This PR’s changes are unrelated — all modifications are in src/Platform/Microsoft.Testing.Extensions.JUnitReport/ and its tests.

Fix

Regenerate the XLF files for MSTest.Analyzers by running:

dotnet msbuild src/Analyzers/MSTest.Analyzers/MSTest.Analyzers.csproj /t:UpdateXlf

Then commit the updated xlf/*.xlf files. This can be done either:

  • On main directly (fixing the base for all open PRs), or
  • In this PR branch if you want to unblock it immediately.

i️ Per the repo guidelines, .xlf files must never be edited by hand — always regenerate via UpdateXlf.

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · sonnet46 34 AIC · ⌖ 5.29 AIC · ⊞ 8K · [◷]( · )

@Evangelink
Evangelink enabled auto-merge (squash) July 28, 2026 12:09
The outcome-mapping table and the XML skeleton both rendered <failure> and
<error> as self-closing while the table also claimed a body, which is an
impossible XML shape. Now that these elements always carry a body, show them
with explicit start/end tags in both places, and correct the <skipped> row:
the writer emits only the message attribute, never a body.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a565c557-8c36-4279-ab7b-41311abcd836
Copilot AI review requested due to automatic review settings July 28, 2026 12:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Medium

@github-actions

This comment has been minimized.

WriteXmlAsync_ErroredTestWithoutException_FallsBackToElementNameForTypeAttribute
verified the element name and the type fallback but left the message attribute
and the body content unasserted, so a regression in either would have gone
undetected by this test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a565c557-8c36-4279-ab7b-41311abcd836
Copilot AI review requested due to automatic review settings July 28, 2026 12:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Medium

@github-actions

Copy link
Copy Markdown
Contributor

🧪 Test quality grade — PR #10286

All 9 new test methods are in JUnitReportFailureBodyTests (new file). The diff in JUnitReportTests.cs only touched helper-method bodies, not [TestMethod]-decorated methods, so those are excluded.

GradeTestMutationNotesHow to improve
A (90–100) new JUnitReportFailureBodyTests.
BuildFailureBody_
WithExceptionTypeMessageAndStackTrace_
MirrorsPrintStackTraceShape
4/4 killed Exact string equality kills all meaningful mutations: type prefix, ": " separator, "\n" join, stack trace.
A (90–100) new JUnitReportFailureBodyTests.
BuildFailureBody_
WithNoExceptionDetails_
ReturnsNull
1/1 killed Null return on all-null input; Assert.IsNull catches any non-null mutation.
A (90–100) new JUnitReportFailureBodyTests.
BuildFailureBody_
WithoutErrorMessage_
OmitsTheSeparator
2/2 killed Exact string confirms no ": " appears and stack trace is still appended with "\n".
A (90–100) new JUnitReportFailureBodyTests.
BuildFailureBody_
WithoutExceptionType_
OmitsTheTypePrefix
2/2 killed Exact string kills the early-return guard mutation and the "\n" join mutation.
A (90–100) new JUnitReportFailureBodyTests.
BuildFailureBody_
WithoutStackTrace_
StillContainsTypeAndMessage
2/2 killed Exact string kills removing type prefix and removing ": " separator in the no-stack-trace path.
A (90–100) new JUnitReportFailureBodyTests.
BuildFailureBody_
WithStackTraceOnly_
ReturnsStackTraceUnchanged
2/2 killed Exact string kills both the "return null" mutation and any wrapper/prefix mutation on the early-return path.
A (90–100) new JUnitReportFailureBodyTests.
WriteXmlAsync_
ErroredTestWithoutException_
FallsBackToElementNameForTypeAttribute
4/4 killed Asserts element name, type fallback, message attribute, and body — all four attributes of the errored outcome.
A (90–100) new JUnitReportFailureBodyTests.
WriteXmlAsync_
FailedTest_
WritesMessageIntoTheFailureBodyAndKeepsAttributes
4/4 killed Asserts element name, both XML attributes, and the full body text — comprehensive for the failure element.
A (90–100) new JUnitReportFailureBodyTests.
WriteXmlAsync_
FailedTestWithoutException_
FallsBackToElementNameForTypeAttribute
2/2 killed Asserts type-attribute fallback to element name and body-only content when no exception type is present.

This advisory comment was generated automatically. Grades are heuristic
and informational — they do not block merging. Re-run with
/grade-tests.

🤖 Automated content by GitHub Copilot. Generated by the Grade Tests on PR (on open / sync) workflow. · sonnet46 91.9 AIC · ⌖ 4.71 AIC · ⊞ 10.3K · [◷]( · )

@Evangelink Evangelink added the state/needs-review Awaiting review from the team. label Jul 28, 2026
@Evangelink
Evangelink merged commit e131c91 into main Jul 28, 2026
32 checks passed
@Evangelink
Evangelink deleted the dev/amauryleve/bookish-bassoon branch July 28, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state/needs-review Awaiting review from the team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an option to include exception messages in Microsoft.Testing.Extensions.JUnitReport failure bodies for GitLab compatibility.

3 participants